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(54) System and method for cellular specimen grading 



(57) The system and method for evaluating the 
amount of marker identifying precipitate in a cellular 
specimen on a microscope siide is disclosed. The auto- 
mated microscope system performs a low magnification 
and high magnification scan of a cellular specimen to 
identify and confirm candidate objects of interest which 
correspond to cells containing a marker identifying pre- 
cipitate. A regularly shaped area centered about a cen- 
troid computed for a candidate object of interest is used 
to define the pixels to be processed. A color ratio is com- 
puted for each pixel (204) and those color ratios indica- 
tive of being dominated by the color corresponding to 
the marker identified precipitate are summed (210) and 
normalized (214). The normalized color ratio is com- 
pared to at least one predetermined threshold to assign 
a grade to the candidate objects of interest (220). The 
grades fora predetermined numberof candidate objects 
of interest are summed (224) to form an aggregate score 
and the aggregate score is compared to a threshold. If 
the aggregate score exceeds a threshold, the candidate 
object of interest is determined to indicate the medical 
condition typically associated with the marker. 



FiC 4 




Q. 
LU 



1 



EP 1 308 712 A2 



2 



Description 

Field of the Invention 

[0001] This invention relates to automated analysis of 
cellular specimens on microscope slides, and more par- 
ticularly, to automated analysis of cellular specimens 
containing stained markers. 

Background of the Invention 

[0002] In the field of medical diagnostics including on- 
cology, the detection, identification, quantitation and 
characterization of cells of interest, such as cancer cells, 
through testing of biological specimens is an important 
aspect of diagnosis. Typically, a biological specimen 
such as bone marrow, lymph nodes, peripheral blood, 
cerebrospinal fluid, urine, effusions, fine needle aspi- 
rates, peripheral blood scrapings or other materials are 
prepared by staining the specimen to identify cells of in- 
terest. One method of cell specimen preparation is to 
react a specimen with a specific probe which can be a 
monoclonal antibody, a polyclonal antiserum, or a nu- 
cleic acid which is reactive with a component of the cells 
of interest, such as tumor cells. The reaction may be 
detected using an enzymatic reaction, such as alkaline 
phosphatase or glucose oxidase or peroxidase to con- 
vert a soluble colorless substrate to a colored insoluble 
precipitate, or by directly conjugating a dye to the probe. 
[0003] For example, substances sometimes known 
as markers may exist in a person's blood as a result of 
some medically abnormal condition. These markers 
may exist for such conditions as preleukemic cancers, 
a trisomy 21 fetus, or other known conditions which 
cause markers to exist in one's blood. These markers 
are normally not visually detectable. However, a blood 
cell sample may be fixed and bound with a substrate of 
an enzyme to produce a colored insoluble precipitate to 
identify the marker. The slides containing the prepared 
cellular specimens are then examined to evaluate the 
amount of the precipitate contained in the cellular spec- 
imen to determine whether the cellular specimen indi- 
cates that the condition exists in the person from which 
the sample was obtained. 

[0004] For example, blood cells are classified into two 
types - red and white cells. The red cells carry oxygen 
in the form of hemoglobin to tissue in a person. The 
white cells are generally related to a person's immunity 
system. White blood cells are comprised of five types of 
which one, neutrophils, has a lobed nucleus which is 
typically used to identify this type of white blood cells. 
In response to the presence of a fetus, the neutrophil 
cells in the blood of a pregnant woman have an elevated 
level of alkaline phosphatase. If the fetus is a trisomy 21 
fetus, the level of alkaline phosphatase in the neu- 
trophils is even higher. Thus, the amount of alkaline 
phosphatase in the neutrophils of a pregnant woman's 
blood may be used as a marker for a trisomy 21 fetus. 



By preparing a blood specimen from a pregnant woman 
with a stain for identifying the alkaline phosphatase in 
neutrophils and a counterstain to facilitate detection of 
the lobed shaped nuclei of neutrophils, a pathologist 
s may visually determine the likelihood that the fetus is a 
trisomy 21 fetus. 

[0005] Examination of biological specimens in the 
past has been performed manually by either a lab tech- 
nologist or a pathologist, in the manual method, a slide 
to prepared with a biological specimen is viewed at a low 
magnification under a microscope to visually locate can- 
didate cells of interest. Those areas of the slide where 
cells of interest are located are then viewed at a higher 
magnification to confirm those objects as cells of inter- 
ns est. The manual method is time consuming and may be 
susceptible to error including missing areas of the slide. 
In the example given above, the low magnification scan 
is performed to identify the neutrophils by the lobed 
shaped nuclei having the counterstain color. 
20 [0006] Automated cell analysis systems have been 
developed to improve the speed and accuracy of the 
slide evaluation process. One known interactive system 
Includes a single high power microscope objective for 
scanning a rack of slides, portions of which have been 
23 previously identified for assay by an operator. In that 
system, the operator first scans each slide at a low mag- 
nification similar to the manual method and notes the 
points of interest on the slide for later analysis. The op- 
erator then stores the address of the noted location and 
30 the associated function in a data fife. Once the points of 
interest have been located and stored by the operator, 
the slide is then positioned in an automated analysis ap- 
paratus which acquires images of the slide at the 
marked points and performs an image analysis. 
35 [0007] There are also known automated specimen 
analysis systems which automatically view slides locat- 
ed in carriers which are loaded in a hopper. The carriers 
are moved, oneatatime, fromthe hopperto a motorized 
XY stage of the microscope. The motorized stage is op- 
40 erated to place one slide in the carrier under the objec- 
tive turret of the microscope and the slide is scanned at 
a low magnification power. The view of the slide through 
the oculars of the microscope is captured by a camera 
which may either be a digital camera or an analog carri- 
es era with an analog/digital (A/D) converter. The digitized 
image is provided to a computer subsystem coupled to 
the automated microscope to detect candidate objects 
of interest on the slide. The information regarding the 
candidate objects of interest is then stored and the view- 
so ing power for the objective turret is increased to a high 
magnification level. The slide is scanned at high magni- 
fication and an image of the slide at the high magnifica- 
tion level is captured by the camera, digitized, and fur- 
ther processed by the computer subsystem to eliminate 
55 debris and other objects which may be part of the cellu- 
larspecimen which do not require analysis. The portions 
of the high magnification image which correspond to 
candidate objects of interest not eliminated by the 
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processing at the high magnification level are then 
stored in a montage for viewing by a pathologist. Infor- 
mation identifying the slide from which the image was 
obtained and the location of the candidate object of in- 
terest on the slide is also recorded with the montage. If s 
the pathologist wants to view the candidate object of in- 
terest on the slide, the pathologist may place the slide 
in a carrier and load it in the automated analysis system. 
The system then moves the slide to a position under- 
neath the objective turret where, under high magnifica- 10 
tion, the pathologist may view the candidate object of 
interest to confirm the selection of the candidate object 
of interest for the montage. 

[0008] One problem with previously known automat- 
ed systems is the difficulty in evaluating the amount of is 
a marker present in a cellular specimen. For example, 
neutrophil alkaline phosphatase (NAP) is typically 
stained to cause the insoluble product which identifies 
the marker to become red in color. The cellular speci- 
men is usually counterstained to make the nuclei of the 20 
cells become blue in color. A pathologist viewing such 
a slide detects neutrophils by locating those cells which 
have a nucleus of the color, shape, and size expected 
for a neutrophil. The pathologist then subjectively eval- 
uates the intensity of the red color for each located neu- 25 
trophil and assigns a score to each one. The pathologist 
then subjectively determines whether the number of in- 
tensely red neutrophils and moderately red neutrophils 
are sufficient to conclude that the cellular specimen is 
indicative of a particular condition. For NAP, the pathol- so 
ogist usually grades the neutrophils with a rating of 0, 
1, 2, 3, or 4 in accordance with a grading scale such as 
the one provided with the procedure for using the rea- 
gent kit sold by Sigma Diagnostics for demonstrating al- 
kaline phosphatase activity in leukocytes. According to 35 
that procedure, a pathologist sums the subjectively as- 
signed ratings for marker identifying precipitate in the 
first 100 neutrophils to arrive at a score which may be 
used to determine the relative red intensity of the neu- 
trophils in the cellular sample. This score may then 40 
be'used to determine whether the condition associated 
with the presence of the marker is indicated. 
[0009] Previously known automated specimen analy- 
sis systems have not been able to provide grading as 
reliable as thai possible with a trained pathologist. For & 
example, U.S. Patent No. 5,352,613 to Tafas et al. dis- 
closes a cytological screening method which purports 
to evaluate the presence and amount of such markers 
as NAP. However, the system of this patent has a 
number of limitations. For one, the perimeter of neu- so 
trophils or other candidate objects of interest must be 
precisely located in the images as the method of this 
patent computes an average optical density for each 
pixel within a candidate object of interest. However, 
where a candidate object of interest, such as a neu- ss 
trophil, is overlapped by a red blood cell, the method of 
this patent may inaccurately define the perimeter of the 
candidate object of interest if it is using the red color 



component to define the perimeter for the candidate ob- 
ject of interest. When the perimeter of the candidate ob- 
ject of interest is not accurately defined, pixels actually 
in a candidate object of interest may be missed and 
those not actually in a candidate object of interest may 
be included in the computation of the density value. In 
fact, this patent does not indicate that the pixels for the 
nuclei of the cells being evaluated are excluded fromthe 
optical density measurements. In some cases, the in- 
clusion of the nuclei pixels or exclusion of pixels actually 
in a cell may skew the measurements. Additionally, the 
method of this patent operates on a single color compo- 
nent of the image of the cellular specimen and, as a re- 
sult, where cells may overlap, the pixel value of the sin- 
gle color component being analyzed may be attenuated 
in the light passed by the overlapped cells. 
[0010] What is needed is a system which grades neu- 
trophils containing NAP as reliably as a trained pathol- 
ogist. What is needed is an automated specimen anal- 
ysis system which does not rely on precise perimeter 
definition of candidate objects of interest in order to 
measure the amount of a marker identifying precipitate 
within a candidate object of interest. What is needed is 
a system which can accurately evaluate the amount of 
a marker identifying precipitate within a cellular speci- 
men even though overlapping cells are present in the 
defined object of interest. 

Summary of the Invention 

[0011] The limitations of previously known automated 
analysis systems and methods may be overcome by a 
system implementing the method of the present inven- 
tion. The inventive method includes the steps of obtain- 
ing a colordigitat image of a magnified viewofa cellular 
specimen bound to a microscope slide, processing the 
color digital image to identify a plurality of candidate ob- 
jects of interest in the cellular specimen, identifying a 
centroid for each candidate object of interest, computing 
a color ratio of at least two color components for each 
pixel of an area centered about each identified centroid, 
computing an average color ratio for all pixels in the area 
centered about each centroid having a computed color 
ratio that exceeds a predetermined color ratiothreshold, 
and comparing the computed average color ratio to at 
least one intensity threshold to evaluate the amount of 
marker identifying precipitate in each area centered 
about each centroid. 

[0012] This method may be implemented by an auto- 
mated analysis system which obtains digital images and 
processes the digital images to locate candidate objects 
of interest in a cellular specimen on a microscope slide. 
In a preferred embodiment, the system computes a ratio 
of two color components for the pixels in a regularly 
shaped area about the identified centroid. In a most pre- 
ferred implementation, the area is a 70 pixel by 70 pixel 
area centered about the centroid. By defining a regularly 
shaped area about the centroid, the grading system no 
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longer needs to traverse an irregular perimeter associ- 
ated with a candidate object of interest to identify the 
pixels required for analysis of the object. Instead, the 
computation of a color ratio using at least two color com- 
ponents effectively includes in the evaluation only those 
pixels which contain evidence of the marker identifying 
precipitate and its relative intensity. Because the area 
centered about the centroid is regularly shaped, the 
processing of the pixels is much faster than previously 
known methods. Additionally, use of the ratio of two 
color components provides more information so the 
method is more accurate than those using only one color 
componentto evaluate the amount of a marker identify- 
ing precipitate. 

[0013] In the preferred implementation of the present 
invention, a 70 pixel by 70 pixel area around a centroid 
is first defined. A ratio of the red pixel intensity to the 
green pixel intensity for each pixel in the area is used 
for the evaluation of NAP in the area. The preferred ratio 
is a ratio of the difference between the red and green 
pixel components to the sum of the red and green pixel 
components. Because the preferred implementation 
does not use the absolute value of the difference, the 
ratio is a signed magnitude within the range of-1 to +1. 
When a cellular sample has been chemically assayed 
with a solution of napthol AS-biphosphate salt and fast 
red violet LB, the presence of alkaline phosphatase is 
indicated by the red color of the precipitate formed by 
hydrolysis. As a result, the preferred color ratios that in- 
dicate the presence of NAP are in the range of slightly 
greater than zero to +1 . Consequently, all of the pixels 
which indicate the presence of NAP within the regularly 
shaped area may be identified by ratios having a posi- 
tive, non-zero value. 

[0014] The color ratios for those pixels having a color 
ratio which is both non-zero and positive are summed 
and then divided by the number of pixels having non- 
zero, positive values. This computed value is a normal- 
ized ratio in the range of zero to +1 . The range fromzero 
to +1 may be divided into five sub-regions which corre- 
spond to the 0, 1,2, 3, and 4 score values commonly 
used to grade the amount of NAP in acellularspecimen. 
The score for 100 of these areas is computed and then 
summed to obtain an aggregate score for the cellular 
specimen. This aggregate score is then compared to a 
threshold to evaluate whetherthe amount of NAP in the 
cellular specimen indicates whether the person from 
which the specimen was obtained has the condition nor- 
mally indicated by the marker. 
[0015] The method and system of the present inven- 
tion provide a number of benefits and advantages. For 
one, because regularly shaped areas are used to eval- 
uate the area about the centroid of a candidate object 
of interest, the processing of the pixels in the area is 
muchfasterthan if an irregularly shaped perimeter must 
be located and then followed to process the image data 
for a candidate object. The use of two color components 
forthe color ratio permits the system and method to use 



a regularly shaped area as the ratio effectively identifies 
only those pixels in the cytoplasm of the neutrophil that 
contain alkaline phosphatase without having to locate 
and traverse an object perimeter. Additionally, the use 
s of two color components allows a system to detect the 
presence of a marker even though debris or an over- 
lapped cell may attenuate the red pixel values in the 
overlapped area. The reader should appreciate that the 
system and method of the present invention may be 
10 used to evaluate the amount of marker in any cellular 
specimen stained to identify the marker present in the 
cellular specimen. These methods include, but are not 
limited to, monoclonal antibodies, polyclonal antibodies, 
in situ hybridization, or reverse transcriptase polymer- 
15 aze chain reaction (RTPCR) methods. 

[0016] Another feature of the present invention is the 
use of a noise reducing filter used forthe color ratio com- 
putation. This filter is a threshold to which the two color 
component difference is compared. If the two color com- 
20 pone nt difference is less than the threshold, the ratio for 
the pixel is not computed. Preferably, this threshold is a 
small positive number. This noise threshold reduces the 
likelihood that a pixel having its two color component 
difference increased to a small positive number by elec- 
ts tronic noise is evaluated for marker identifying precipi- 
tate. This filter also helps exclude pixels from a red blood 
cell which extends beyond an overlap area with a can- 
didate object of interest. Thus, the filter reduces the 
need to accurately define the perimeter of the candidate 
30 object of interest and compensates for electronic noise 
in the generation of the image data. 
[0017] These and other advantages and benefits of 
the present inventive system and method may be ascer- 
tained from the detailed description of the invention. 

35 

Brief Description of the Drawings 

[0018] The accompanying drawings, which are incor- 
porated and constitute a part of the specification, illus- 
40 trate preferred and alternative embodiments of the 
present invention and, together with a general descrip- 
tion given above and the detailed description of the em- 
bodiments given below, serve to explain the principles 
of the present invention. 

45 

Fig. 1 is a perspective view of an apparatus for au- 
tomated cell analysis embodying the present inven- 
tion; 

Fig. 2 is a block diagram of the apparatus shown in 
so Fig. 1; 

Fig. 3 is a block diagram of the microscope control- 
ler of Fig. 2; 

Fig. 4 is a flowchart of the preferred method for 
grading objects in an image of a cellular specimen 
55 obtained by the apparatus of Fig. 1 ; 

Fig. 5 is a perspective view of a regularly shaped 
area in an image of a cellular specimen processed 
by the exemplary method depicted in Fig. 3; and 
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Fig. 6 is a block diagram of a cellular specimen 
grading system implementing the method of Fig. 4. 

Detailed Description of the Invention 

[0019] Referring now to the figures, an apparatus for 
automated cell analysis of biological specimens is gen- 
erally indicated by reference numeral 10 as shown in 
perspective view in Fig. 1 and in block diagram form in 
Fig. 2. The apparatus 10 comprises a microscope sub- 
system 32 housed in a housing 12. The housing 12 in- 
cludes a slide carrier input hopper 16 and a slide carrier 
output hopper 18. A door 14 in the housing 12 secures 
the microscope subsystem from the external environ- 
ment. A computer subsystem comprises a computer 22 
having a system processor 23, an image processor 25 
and a communications modem 29. The computer sub- 
system further includes a computer monitor 26 and an 
image monitor 27 and other externa! peripherals includ- 
ing storage device 21, track ball device 30, keyboard 28 
and color printer 35. An external power supply 24 is also 
shown for powering the system. Viewing oculars 20 of 
the microscope subsystem project from the housing 12 
for operator viewing. The apparatus 10 further includes 
a CCD camera 42 for acquiring images through the mi- 
croscope subsystem 32. Preferably, a switch is included 
in the system which either provides the magnified view 
from the objective turret 44 to oculars 20 or camera 42. 
A microscope controller 31 under the control of system 
processor 23 controls a number of microscope-subsys- 
tem functions described further in detail. An automatic 
slide feed mechanism 37 in conjunction with X-Y stage 
38 provide automatic slide handling in the apparatus 1 0. 
An illumination light source 48 projects light onto theX-Y 
stage 38 which is subsequently imaged through the mi- 
croscope subsystem 32 and acquired through CCD 
camera 42 for processing in the image processor 25. A 
Z stage or focus stage 46 under control of the micro- 
scope controller 31 provides displacement of the micro- 
scope subsystem in the Z plane for focusing. The mi- 
croscope subsystem 32 further includes a motorized ob- 
jective turret 44 for selection of objectives. 
[0020] The purpose of the apparatus 10 is for the un- 
attended automatic scanning of prepared microscope 
slides for the detection of candidate objects of interest, 
such as particular cells which may contain marker iden- 
tifying precipitate, and evaluation of the amount of a 
marker identifying precipitate in the detected candidate 
objects of interest. Apparatus 10 automatically locates 
candidate objects of interest present in a biological 
specimen on the basis of color, size and shape charac- 
teristics. Grades indicative of the amount of marker 
identifying precipitate for the candidate objects of inter- 
est are determined and summed to generate a score for 
the biological specimen. This score may be used to eval- 
uate whether the biological specimen is indicative of a 
medical condition typically associated with the marker 
that produced the marker identifying precipitate. A 



number of stains and countersigns are used to produce 
colored marker identifying precipitate in various cells 
and cell structures of the biological specimen. The ap- 
paratus of the present invention is used to detect this 
s precipitate to identify candidate objects of interest. 
[0021] During operation of the apparatus 10, a pathol- 
ogist or laboratory technologist mounts prepared slides 
onto slide carriers. A slide carrier holds up to 4 slides 
and up to 25 slide carriers may be loaded into input hop- 
10 per 16. The operator can specify the size, shape and 
location of the area to be scanned or alternatively, the 
system can automatically locate scan areas. The oper- 
ator then commands the system to begin automated 
scanning of the slides through a graphical user inter- 
ns face. Unattended scanning begins with the automatic 
loading of the first carrier onto precision motorized X-Y 
stage 38. A bar code label affixed to the slide is read by 
abarcode reader 33 during this loading operation. Each 
slide is then scanned at a user selected low microscope 
20 magnification, for example, 20x, to identify candidate 
objects based on their color, size and shape character- 
istics. TheX-Y locations of candidate objects are stored 
until scanning is completed. 

[0022] After the low magnification scanning is com- 

25 pleted, the apparatus automatically returns to each can- 
didate object, focuses at a higher magnification, such 
as 60x for NAP evaluation, and captures a digitized im- 
age for further analysis to confirm the object candidate. 
The centroid for each confirmed cell candidate is com- 

30 puted and stored for evaluation of the marker identifying 
precipitate. Apparatus 10then returns to the centroid for 
the first confirmed candidate object of interest and cap- 
tures a color image of an area centered about the cen- 
troid. The pixel data for this area is processed to deter- 

35 mine the amount of marker identifying precipitate in the 
area and a grade is assigned to the object. Apparatus 
10 continues processing and grading areas centered 
about other confirmed candidate objects of interest until 
a predetermined number of objects have been proc- 

40 essed. An aggregate score is then computed from the 
grades for the predetermined number of objects. The 
object grades, aggregate score and images may then 
be stored to a storage device 21 such as a removable 
hard drive or DAT tape or transmitted to a remote site 

43 for review or storage. The stored images for each slide 
can be viewed in a mosaic of images for further review. 
In addition, the pathologist or technologist can also di- 
rectly view a detected cell through the microscope using 
oculars 20 or image monitor 27. 

so [0023] Referring to Fig. 3, microscope controller 31 is 
shown in more detail. Microscope controller 31 includes 
a number of subsystems connected through a system 
bus. System processor 102 controls these subsystems 
and is controlled by apparatus system processor 23 

35 through an RS 232 controller 110. System processor 
102 controls a set of motor control subsystems 114 
through 124 which control the input and output feeder, 
motorized turret 44, X-Y stage 38, and Z stage 46 (Fig. 
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2). Histogram processor 108 receives input from CCD 
camera 42 for computing variance data during the fo- 
cusing operation described further herein. 
[0024] System processor 102 further controls illumi- 
nation controller 106 for control of substage illumination 
48. The light output from the halogen light bulb which 
supplies illumination for the system can vary over time 
due to bulb aging, changes in optical alignment, and oth- 
er factors. In addition, slides which have been "over 
stained" can reduce the camera exposure to an unac- 
ceptable level. In orderto compensate forthese effects, 
illumination controller 106 is included. This controller is 
used in conjunction with light control software to com- 
pensate for the variations in light level. The light control 
software samples the output from the camera at inter- 
vals (such as between loading of slide carriers), and 
commands the controller to adjust the light level to the 
desired levels. In this way, light control is automatic and 
transparent to the user and adds no additional time to 
system operation. 

[0025] System processor 23 is preferably an IBM 
compatible PC with an Intel Pentium 90 MHz processor, 
32MB of RAM, and two 1GB hard drives with one hard 
drive being removable. The operating system for system 
processor 23 is Windows for Workgroups 3.1 available 
from Microsoft Corporation of Redmond, Washington. 
Image processor 25 is preferably a Matrox Imaging Se- 
ries 640 board set available from Matrox Electronics 
Systems, Ltd. of Dorval, Quebec, Canada. The pre- 
ferred image processor is provided with support soft- 
ware and the Matrox Imaging Library (MIL). Microscope 
controller system processor 102 is an Advanced Micro 
Devices AMD29K device. 

[0026] The low magnification image processing iden- 
tifies cells having a nucleus which is stained a particular 
color which corresponds to the stain or counterstain 
used to prepare the cellular specimen. For example, a 
blood smear used to evaluate the presence of alkaline 
phosphatase in neutrophils is typically stained with a so- 
lution of napthol AS-biphosphate salt and fast red violet 
LB. The presence of alkaline phosphatase in a neu- 
trophil (NAP) in the cellularspecimen is indicated by the 
red color of the precipitate formed by hydrolysis. The 
specimen is usually counterstained with a counterstain 
such as hematoxylin to produce a blue insoluble precip- 
itate in white blood cell nuclei. The resulting specimen 
from this stain/courtterstain procedure results in white 
blood cells which may be identified by the cells having 
blue nuclei. These cells may be processed to identify 
neutrophil cells based on the shape and size of the nu- 
clei. Thus, the low magnification processing identifies 
candidate objects of interest, such as neutrophils, from 
the color, shape, and size of objects in the image of the 
cellular specimen. 

[0027] The processing performed by the system dur- 
ing the high magnification image processing further 
evaluates color, shape and size of the ceil nucleus for 
each of the candidate objects of interest to eliminate ob- 



jects not likely to contain the marker identifying precip- 
itate. In a preferred implementation of the present inven- 
tion for NAP, the minimum area for identifying candidate 
objects of interest is 14 urn 2 . The preferred compact- 

5 ness value is for objects having a nucleus which isgreat- 
erthan the value 1.25. Compactness is a term indicating 
the shape of the perimeter of the nucleus which is well 
known in the field. The centroid of the identified candi- 
date object of interest is then computed and the centroid 

10 is stored forthe score processing. After a predetermined 
number of candidate objects of interest are confirmed 
and their corresponding centroids stored, score 
processing is performed. 

[0028] In the preferred implementation, the image 
is processing performed at the low magnification and high 
magnification levels is that which is disclosed in our co- 
pending patent application entitled METHOD AND AP- 
PARATUS FOR AUTOMATED IMAGE ANALYSIS OF 
BIOLOGICAL SPECIMENS, Serial No. 08/758436 and 
20 filed on November 27, 1 996. The disclosure of that ap- 
plication is hereby expressly incorporated by reference. 
The color conversion, low pass filtering, thresholding, 
and morphological processing disclosed in that docu- 
ment is preferably used to identify candidate objects of 
25 interest. The centroid and morphological characteristics 
for a candidate object of interest, such as its size and 
compactness, is obtained from functions in the MIL for 
the preferred image processor. Although this is the pre- 
ferred method for identifying and confirming candidate 
30 objects of interest in the image of a cellular specimen, 
other known methods may be used as long as the meth- 
od identifies the location and general shape of those 
cells which contain the marker identifying precipitate. 
[0029] The method of the score processing is shown 
35 in Fig. 4. That process begins by retrieving the centroid 
for the first confirmed object of interest and captures the 
color pixel values for a regularly shaped area centered 
about the centroid (Block 200). The area may be any 
regularly shaped area that is easily traversed by known 
io programming loop control such as two dimensional par- 
allelograms. The most preferred regular shape is a 
square and the most preferred size of the square is 70 
pixels x 70 pixels. The regularly shaped area should be 
sufficiently large enough to include all of the pixels con- 
45 taining image data forthe marker identifying precipitate 
that are associated with a candidate object of interest. 
For example, the most preferred 70 pixels x 70 pixels 
shape corresponds to an area large enough to cover the 
cytoplasm and nucleus of the cell corresponding to a 
so confirmed candidate object of interest. The most pre- 
ferred area is shown in Fig. 5. As can be seen from the 
figure, the regular shaped area extends beyond the pe- 
rimeter of the candidate object of interest so that the ar- 
ea centered about the centroid of the object of interest 
55 includes all of the pixels of the candidate object of inter- 
est. The reader should appreciate that any area which 
may be used to simplify evaluation of the amount of 
marker identifying precipitate in a candidate object of 
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interest without having to define the perimeter of the 
candidate object of interest in order to identify the can- 
didate object of interest pixels to be included in the eval- 
uation is within the principles of the present invention. 
[0030] A color ratio for each pixel is then computed 
(Block 204). In the preferred implementation, the color 
ratio is the difference of two pixel color components to 
the sum of the same two pixel color components. In the 
most preferred implementation, the red and green pixel 
color components are used to score NAP in a cellular 
specimen stained with a solution of napthol AS-biphos- 
phate salt and fast red violet LB and then counterstained 
with hematoxylin. Thus, the ratio may be expressed as 
(R-G)/ (R+G), although other color component combi- 
nations may be used. The color component selection 
preferably includes the two color components which 
best correspond to the marker identifying participate 
color (typically stain color) and the cell identifying color 
(typically counterstain color). The ratio identifies those 
pixels which are dominated by the first color component 
(red in the most preferred embodiment) from which the 
second color component (green in the most preferred 
embodiment) is subtracted. The ratio lies in the range 
of-1 to +1 where the ratio is positive for pixels dominated 
by the first color component, negative for pixels domi- 
nated by the second color component, and zero where 
the two components are equal 
[0031] Use of the color ratio permits the system and 
method of the present invention to process the pixels of 
the regularly shaped area to evaluate the amount of 
marker identifying precipitate without having to traverse 
the perimeter to identify all of the pixels in a candidate 
object of interest. Instead, the ratio of two color compo- 
nents, at least one of which corresponds to the color of 
the marker identifying precipitate, allows the system to 
identify the pixel values corresponding to the marker 
identifying precipitate without having to reference the 
geometry of the candidate object of interest. This per- 
mits the method of the present invention to evaluate 
each candidate object of interest more quickly and more 
accurately than methods that rely on geometry to iden- 
tify pixels corresponding to the marker identifying pre- 
cipitate. 

[0032] Most preferably, a color ratio is only computed 
forthose pixels having a two color component difference 
which is larger than a predetermined noise threshold. 
Preferably, the noise threshold is a small positive 
number and, most preferably, the threshold is five (5). 
The effect ofthe noise threshold is to reduce the number 
of pixels in a white or dark area which may appear to 
have color indicative of the marker identifying precipi- 
tate. White or black pixels should have a value in which 
all ofthe color components are zero (0) or two hundred 
and fifty-five (255), respectively. As a result, a two color 
component difference for a white or black pixel should 
be zero and the pixel not included in the color ratio nor- 
malization discussed below. However, electronic noise 
may cause a pixel that should be white to have a color 



component with a small, positive value or a pixel that 
should be black to have a color component that is slight- 
ly less than 255. If that color component with the noise 
induced value is the red component for the white pixel 
5 or the green component for the black pixel in the pre- 
ferred two color component difference for NAP grading, 
then a small, positive ratio is computed for both the white 
and black pixel. These color ratios would then be includ- 
ed in the color ratio normalization and skew the results. 
10 This skewed ratio may then affect the grade for candi- 
date object of interest and possibly the score forthe cel- 
lular specimen. However, the noise threshold prevents 
the computation of the color ratio for the pixel as long 
as the two color component difference is less than the 
15 noise threshold. Additionally, the noise threshold reduc- 
es the likelihood that pixels from an overlapped cell are 
included in the grading and scoring ofthe cellular spec- 
imen. For example, pixels for a red blood cell that over- 
laps a candidate object of interest and which extends 
20 beyond the perimeter ofthe candidate object of interest 
are, typically, not intensely red. As a consequence, the 
noise threshold would eliminate those pixels ofthe red 
blood cell outside the candidate object of interest that 
are nominally red colored from the candidate object 
25 grading. This filtration is done without reference to the 
perimeter of the candidate object of interest. 
[0033] Alternatively, pixels which are almost white or 
almost black may be considered white or black so they 
are not used for grading a candidate object of interest. 
30 This method determines whether the color component 
having the largest value is less than a white threshold 
or whetherthe color component having the smallest val- 
ue is greater than a black threshold. For example, using 
a white threshold of 1 0, an almost white pixel having red, 
35 green and blue values of 9, 3, and 8, respectively, would 
be considered a white pixel with a two color component 
difference of zero (0) even though the actual red-green 
difference is six (6). Thus, a noise threshold of 5 would 
have included the color ratio for the pixel in the object 
40 grading but the white threshold of 10 would remove it 
from the grading. Likewise, a black threshold of two hun- 
dred and fifty (250) would classify pixels having a small- 
est color component of 250 or higher as black pixels and 
not compute a color ratio forthe pixel. 
45 [0034] The process of Fig. 4 then continues by sum- 
ming all ofthe color ratios forthe pixels having a positive, 
non-zero color ratio and dividing the sum by the number 
of pixels used to compute the sum (Block 208, 21 0, 212, 
214). This normalized color ratio forthe area is then 
so compared to at least one threshold to determine the 
amount of marker identifying precipitate (Block 220). In 
the preferred embodiment for NAP for the stain and 
counterstain discussed above, there are fourth resholds 
forthe range of 0to+1. These fourthresholds divide the 
55 range into five sub-ranges which correspond to the five 
grades typically used to classify the intensity ofthe NAP 
precipitate. The grade forthe area is then added to an 
object score (Block 224) and the process continues until 
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a predetermined number of candidate objects of interest 
have been processed (Block 228). In the most preferred 
embodiment, the predetermined number of candidate 
objects of interest to evaluate the amount of NAP in a 
cellular specimen is one hundred (100) objects. The ob- 
ject score is then stored in association with the slide 
(Block 230) for later evaluation by a pathologist. Alter- 
natively, the object score may be compared to a prede- 
termined threshold which indicates whether the cellular 
specimen indicates the condition associated with the 
marker. 

[0035] In the preferred embodiment of the present in- 
vention, the process of Fig. 4 is implemented in a C pro- 
gram which is downloaded to image processor 25 for 
execution, although other programming languages may 
be used. Additionally, the program may be executed on 
another processor of the system as long as the image 
data stored in image processor 25 is available for the 
program and/or hardware implementing the process of 
Fig. 4. 

[0036] A system which may be used to implement the 
preferred method of the present invention is shown in 
Fig. 6. System 150 processes image data stored in im- 
age processor 25 to generate a score for a cellular spec- 
imen bound to a slide. Color ratio generator 154 com- 
putes the color ratios for pixels within a regularly defined 
area about a centroid for a candidate object of interest 
in the image data. Preferably, color ratio generator 154 
includes a noise threshold to determine for which pixels 
a color ratio is computed. The computed color ratios are 
provided to color ratio comparator 158 which compares 
the color ratio to a positive, non-zero threshold to deter- 
mine whether the color ratio is included in the normal- 
ized color ratio for the specimen. Those color ratios 
passed by comparator 158 are provided to color ratio 
normalizer 160 which sums the color ratios passed to it 
and divides by the number of color ratios passed to it. 
The normalized color ratio for a candidate object of in- 
terest is provided to grade generator 164 which com- 
pares the normalized color ratio to one or more thresh- 
olds and assigns a grade to the candidate object of in- 
terest. The assigned grade is provided to score gener- 
ator 168. Score generator 168 sums the grades for a 
predetermined number of candidate objects of interest 
to generate an aggregate score which may be stored 
with data for the slide. The score may be used to deter- 
mine whether the cellular specimen indicates a particu- 
lar medical condition. The components of system 150 
may be implemented in hardware or software or a com- 
bination thereof. 

[0037] In use, laboratory technologists prepare a plu- 
rality of slides with cellular specimens which are treated 
to make a marker visually detectable. The slides are 
then loaded onto slide carriers and placed in the slide 
hopper of the automated microscope system. The sys- 
tem is initialized and slide carriers are fed to the motor- 
ized stage beneath the objective turret. The low magni- 
fication and high magnification processing is performed 



to identify and confirm candidate objects of interest. Im- 
ages of a regularly shaped area centered about the cen- 
troid of a predetermined number of confirmed candidate 
objects are then graded using the method of the present 
5 invention. The aggregate score forthe cellular specimen 
on the slide is then stored with an identifier for the slide 
and the next slide on the carrier is processed. Each slide 
on each carrier is processed until all slides in all of the 
carriers have been processed. Atthe end of the process, 
10 a montage ofthe confirmed candidate objects of interest 
for each slide, a grade for each object, a location for 
each object, and an aggregate score for each slide are 
stored in the computersubsystem. This information may 
be used by a pathologistto determine whether a cellular 
is specimen indicates a specific medical condition orto re- 
view the objects on the slide which were graded. 
[0038] While the present invention has been illustrat- 
ed by the description of the preferred and alternative 
embodiments and while the embodiments have been 
20 described in considerable detail, it is notthe intention of 
the applicant to restrict or anyway limit the scope ofthe 
appended claims to such detail. Additional advantages 
and modifications will readily appear to those skilled in 
the art. The invention's broader aspects are therefore 
25 not limited to the specific details, representative appa- 
ratus and method, or illustrative examples shown and 
described. Accordingly, departures may be made from 
such details without departing from the spirit or scope 
of applicant's general inventive concepts. 



Claims 

1. A system for identifying an object of interest in a bi- 
ological sample, comprising: 

a microscope subsystem comprising an auto- 
mated stage, an automated objective turret, 
one or more objectives, and a light source; 
a camera; and 

a computer program on a computer readable 
medium; 

wherein the camera is adapted to acquire an 
image of a biological sample through at least one 
objective ofthe microscope subsystem that is illu- 
minated by the light source and the computer pro- 
gram comprises instructions that cause the system 
to process the biological sample according to the 
steps of 

performing a low magnification scan ofthe bio- 
logical sample at a plurality of locations; 
obtaining an image at each of the plurality of 
locations at low magnification; 
processing the low magnification images to 
identify candidate objects of interest; 
storing a location associated with each identi- 
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fied candidate object of interest; 
adjusting the magnification objective to a high- 
er magnification; 

obtaining a higher magnification image at each 
stored location for each identified candidate ob- 
ject of interest; and 

processing the higher magnification image to 
identify an object of interest. 

2. The system of claim 1 further comprising a proces- 
sor upon which the computer program is stored. 



centroid for a candidate object of interest, com- 
putes a color ratio of at least two color compo- 
nents for each pixel of each centroid, computes 
an average color ratio for all pixels in the cen- 
s troid having a computed color ratio that ex- 

ceeds a predetermined color ratio threshold 
and compares the computed average color ra- 
tio to at least one intensity threshold. 

10 7. The apparatus of claim 6, wherein the motorized 
stage comprises X-and Y-motors. 



The system of claim 1 wherein the computer pro- 
gram further comprises instructions that cause the 
system to implement the steps of processing the 
higher magnification images and/or lower magnifi- 
cation images by: 



8. The apparatus of claim 6, wherein the light source 
is located below the stage. 



The apparatus of claim 6, wherein the camera is a 
CCD c 



computing a color ratio of at least two color 
components for each pixel of an image; 20 
computing an average color ratio for all pixels 
in the image having a computer color ratio that 
exceeds a predetermined ratio threshold; and 
comparing the computed average ratio to at 
least one intensity threshold to evaluate the 25 
amount of detectable marker in each image. 

4. The system of claim 3 wherein the color ratio is com- 
puted as a ratio of red pixel components to green 
pixel components. 30 

5. The system of claim 3 wherein the color ratio is com- 
puted as a ratio of a difference between a first pixel 
color component and a second pixel color compo- 
nent to a sum of first pixel color component and the 35 
second pixel color component. 

6. An apparatus for automated analysis of a biological 
specimen, comprising: 

40 

a microscope system comprising 
a motorized stage; 

a light source located such that light from 
the light source illuminates a sample on the mo- 
torized stage; 4S 

a plurality of objectives located above the 
stage; and 

a motorized objective turret, 
a camera in optical communication with a sam- 
ple on the motorized stage; so 
a computer comprising a processor in electrical 
communication with the microscope system 
and the camera; and 

a computer program on computer readable me- 
dium comprising instructions that control the 55 
microscope system, the camera, and compris- 
ing an algorithm to process images acquired by 
the camera, wherein the algorithm identifies a 



10. A method for evaluating the amount of a visually de- 
tectable marker identifying precipitate in a cellular 
specimen bound to a microscope slide comprising 
the steps of: 

obtaining a color digital image of a magnified 
view of a cellular specimen bound to micro- 
scope slide; 

processing the color digital image to identify a 
plurality of candidate objects of interest in the 
cellular specimen; 

identifying a centroid for identified candidate 
objects of interest in the plurality of candidate 
objects of interest; 

computing a color ratio of at least two color 
components for each pixel of an area centered 
about each identified centroid; 
computing an average color ratio for all pixels 
in the area centered about each centroid that 
have a computed color ratio that exceeds a pre- 
determined color ratio threshold; and 
comparing the computed average color ratio to 
at least one intensity threshold to evaluate the 
amount of marker identifying precipitate in each 
area centered about each centroid. 

11. The method of claim 10, the color ratio computing 
step further comprising the steps of: 

computing a color ratio of a red pixel compo- 
nent to a green pixel component for each pixel. 

12. The method of claim 10, the color ratio computing 
step further comprising the step of: 

computing a color ratio of a difference between 
a first pixel color component and a second pixel 
color component to a sum of the first pixel color 
component and the second pixel color compo- 
nent for each pixel in the area. 
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13. The method of claim 12, wherein the first pixel color 
component is red and the second pixel color com- 
ponent is green. 

14. The method of claim 12, wherein said comparing 
step compares the computed average color ratio is 
compared to a plurality of ordered thresholds to de- 
termine a grade forthe amount of marker identifying 
precipitate in each area centered about each cen- 
troid. 

15. The method of claim 14, further comprising the step 
of: 

summing the grades for a plurality of areas to 
determine a score forthe cellular specimen. 

16. The method of claim 15 wherein the cellular speci- 
men score is the sum of the grades for one hundred 
areas centered about one hundred centroids iden- 
tified in the cellular specimen. 

17. The method of claim 12 further comprising the step 
of: 

comparing the two color component difference 
to a noise threshold; and 
computing the color ratio in response to the two 
color component difference being greater than 
the noise threshold. 

18. The method of claim 12 further comprising the step 
of: 

comparing a color component for a pixel having 
a largest value to a white threshold and a color 
component forthe pixel having a smallest value 
to a black threshold to determine whether a 
color ratio for the pixel is computed. 

19. A system for evaluating the amount of a visually de- 
tectable marker identifying precipitate in a cellular 
specimen bound to a microscope slide comprising: 

an image processor for obtaining a color digital 
image of a magnified view of a cellular speci- 
men bound to microscope slide; 
means for identifying candidate objects of inter- 
est in the cellular specimen; 
means for identifying a centroid for each can- 
didate object of interest; 
means for computing a color ratio of at least two 
color components for each pixel of an area cen- 
tered about each identified centroid; 
means for computing an average color ratio for 
all pixels in the area centered about each cen- 
troid that have a computed color ratio that ex- 
ceeds a predetermined ratio threshold; and 



means for comparing the computed average 
ratio to at least one intensity threshold to eval- 
uate the amount of marker identifying precipi- 
tate in each area centered about each centroid. 

s 

20. The system of claim 17 wherein said means for 
computing a color ratio computes a ratio of red pixel 
component to a green pixel component 

10 21. The system of claim 17 wherein said means for 
computing a color ratio computes a ratio of a differ- 
ence between a first pixel color component and a 
second pixel color component to a sum of the first 
pixel color component and the second pixel color 

is component. 

22. The system of claim 19 wherein the first pixel color 
component is red and the second pixel color com- 
ponent is green. 

20 

23. The system of claim 19 wherein said means for 
comparing the computed average color ratio com- 
pares the computed average color ratio to a plurality 
of ordered thresholds to determine a grade for the 

23 amount of marker in each area centered about each 
centroid. 

24. The system of claim 21 further comprising: 

30 means forsumming the grades for a plurality of 

areas to determine a score forthe cellularspec- 
imen. 

25. The system of claim 22 wherein said means for 
35 summing sums the grades for one hundred areas 

centered about one hundred centroids identified in 
the cellular specimen. 

26. A method for evaluating the amount of a visually de- 
40 tectable marker identifying precipitate in a plurality 

of candidate objects of interest located in a digital 
image of a cellular specimen bound to a microscope 
slide comprising the steps of: 

45 identifying a centroid for each candidate object 

of interest in a plurality of candidate objects of 
interest; 

computing a color ratio of at least two color 
components for each pixel of an area centered 

50 about each identified centroid; 

computing an average color ratio for all pixels 
in the area centered about each centroid that 
have a computed color ratio that exceeds a pre- 
determined color ratio threshold; and 

55 comparing the computed average color ratio to 

at least one intensity threshold to evaluate the 
amount of marker identifying precipitate in each 
area centered about each centroid. 
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27. The method of claim 10 wherein the area centered 
about a centroid is a regularly shaped area. 

28. The method of claim 10 wherein the regularly 
shaped area is a parallelogram. 

29. The method of claim 10, the color ratio computing 
step further comprising the step of: 

computing a color ratio of a difference between 
a first pixel color component and a second pixel 
color component to a sum of the first pixel color 
component and the second pixel color compo- 
nent for each pixel in the area. 

30. The method of claim 12 wherein the first pixel color 
component is red and the second pixel color com- 
ponent is green. 



specimen; 

a color ratio comparatorforcomparing the com- 
puted color ratios to a predetermined threshold 
to determine which color ratios are included in 
a normalized color ratio; 
a color ratio normaiizer for generating a normal- 
ized color ratio; 

a grade generatorfor generating a grade forthe 
candidate object of interest from the normalized 
color ratio; and 

a score generator for generating a score indic- 
ative of an amount of a marker identifying pre- 
cipitate in the cellular specimen by summing 
grades for a predetermined number of candi- 
date objects of interest. 

. The system of claim 35 said color ratio generator 
further comprising: 



31. The method of claim 12 wherein said comparing 
step compares the computed average color ratio is 
compared to a plurality of ordered thresholds to de- 
termine a grade forthe amount of marker identifying 
precipitate in each area centered about each cen- 
troid. 



a noise threshold used by the color ratio gen- 
eratorto determine whether a color ratio is com- 
puted for a pixel. 

37. The system of claim 35 said color ratio generator 
further comprising: 



32. The method of claim 15 further comprising the step 



summing the grades for a plurality of areas to 30 
determine a score forthe cellular specimen. 



a white threshold and a biack threshold which 
are used by the color ratio generator to deter- 
mine whethera color ratio is computed for a pix- 



33. The method of claim 29 further comprising the step 
oft 

comparing the two color component difference 
to a noise threshold; and 
computing the color ratio in response to the two 
color component difference being greater than 
the noise threshold. 

34. The method of claim 29 further comprising the step 



comparing a color component for a pixel having 
a largest value to a white threshold and a color 
component forthe pixel having a smallest value 
to a black threshold to determine whether a 
color ratio for the pixel is computed. 

35. A system for scoring the amount of a marker iden- 
tifying precipitate in a cellular specimen bound to a 
microscope slide comprising: 



a color ratio generator for computing color ra- 
tios for pixels within an area centered about a 
centroid for a candidate object of interest in im- 
age data of a magnified view of the cellular 



EP 1 308 712 A2 




EP1 308 712 A2 



27 



COLOR 
PRINTER 



COMPUTER 
MONITOR 




IMAGE 
MONITOR 






0 0 



KEYBOARD <^Z^> 



; PARALU 
"35 



IMAGE 
PROCESSOR 



25 




FiC 2 



EP 1 308 712 A2 



;OI§ <x> 



5^ 



mo 

cp5 



/ 



fO 



5d 



SO 11 <x> 



E V 
hi tr 

CD 2 



OI 



tilH 

<o 

x i 

2 



OS 



18 

1 



o 



as 



<=> 



— u 

"J DC 

co b 



OI 



so 



EP 1 308 712 A2 



200. 



^ 

DEFINE AREA 
TO GRADE 



204 




ADO COLOR RATIO 
TO AGGREGATE 
SCORE 




COfvPARE 
NORMALIZED SCORE 
TOTWESHOLDIS) 
AND GRADE 



224. 



YES 



FIG 4 




STORE COMPOS- 
ITE GRADE WITH 
SLIDE DATA 



EP 1 308 712 A2 




25 N 



IMAGE 
PROCESSOR 



154^ 



COLOR RATIO 
GENERATOR 



/5£L 



COLOR RATIO 
COMPARATOR 



FIC5 



164^ 



^ 



GRADE 




SCORE 


GENERATOR 




GENERATOR 



/60 N 



COLOR RATIO 
NORMAUZER 



rice 



